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nRSCRIPTlON 

Method and system for control of a device. 

This invention relates in general to a method and system for control of a device and, in 
particular, to a method and system for remote control of a consumer electronics device, 
5 e.g. audio-visual device such as television (TV), video cassette recorder (VCR), digital 
video disc (DVD) player, personal computer (PC) etc. 

Remote controls are used today together with almost any consumer electronics device, 
e.g. television, DVD player, tuner, etc. In the average household, multiple remote 
10 controls - often one for each consumer electronics device - can be required. Even for a 
person well acquainted with the consumer electronics devices he owns, it is a challenge 
to remember what each button on each remote control is actually for. Furthermore, the 
on-screen menu-driven navigation available for some consumer electronics devices is 
often less than intuitive, particularly for users that might not possess an in-depth 
15 knowledge of the options available for the device. The result is that the user must 

continually examine the menu presented on the screen to locate the option he is looking 
for, and then look down at the remote control to search for the appropriate button. Quite 
often the buttons are given non-intuitive names or abbreviations. Additionally, a button 
on the remote control might also perform a further function which is accessed by first 
20 pressing a mode button. The multitude of options available for modem consumer 

electronics devices unfortunately mean that for many users, programming such a device 
can become an exercise in frustration. The large number of buttons and non-intuitive 
menu options can make the programming of a device more difficult than necessary and 
often result in the user not getting the most out of the devices he has bought 

25 

Using all one's consumer electronics devices to me full is made even more difficult by 
the fact that almost every consumer electronics device today comes with its own remote 
control device. Whilst most remote control button abbreviations and symbols are by 
now standardised to allow marketing of the same remote control device in countries of 
30 different languages, even so it might be that different abbreviations or symbols are used 
on different remote controls to perform the same function, for example the abbreviation 



-2- 



PHDE030423 EPP 



"CH" and "PR" might be used to indicate "channel" or "program", meaning essentially 
the same thing. The remote controls also differ in shape, size, overall appearance and 
even battery requirements. 

5 In an effort to reduce the confusion caused by such a multitude of remote controls, a 
new product category of "universal remote controls" has been developed. However, 
even a universal remote control cannot hope to access all the functions offered by every 
consumer electronics device available on the market today, particularly since new 
technologies and features are continually being developed. Furthermore, the wide 
10 variety of functions offered by modern consumer electronics devices necessitates a 
correspondingly large number of buttons to invoke these functions, requiring an 
inconveniently large remote control to accommodate all the buttons. 

A number of developments attempt to address the problem of simple remote control of 
15 consumer electronics devices. For example EP 0372674 suggests a method involving a 
camera and a stationary light source, both connected to a microcomputer. The moveable 
camera, containing a lens and a detector, is mounted on the head of the user. 
Movements of the user's head relative to the fixed light source, detected by the detector 
in the camera, are translated into movements of a cursor on a television screen. An 
20 obvious disadvantage of such a method is that the user has to wear a camera on his 
head. He must move his head in various directions in order to move the cursor on the 
screen, whilst continually watching the screen to see if he is directing the cursor to the 
desired position. 

25 Therefore, an object of the present invention is to provide a method and user interface 
for convenient and intuitive remote control of a device. 

To this end, the present invention provides a method for control of a device, which 
method comprises visually presenting a number of user options for the device to be 
30 controlled, aiming a pointing device comprising a camera at the visual presentation of 
the user options to choose a desired option, generating an image of a target area aimed 
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at by the pointing device, and comparing the target area image with a pre-defined 
template of the visual presentation to determine the chosen option. The phrase "target 
area image" is to be understood in the broadest possible sense, for example the target 
5 area image might comprise merely image data concerning significant points of the 
entire image, e.g. enhanced contours, corners, edges etc. 

An appropriate user interface for remote control of a device comprises an accessing unit 
for accessing pre-defined templates associated with visual presentations of user options 
10 for the device to be controlled, a pointing device for aiming at a desired option in a 

visual presentation of the user options, comprising a camera for generating an image of 
a target area of at least part of the visual presentation, and an image interpreter for 
locating the target area or a point of the target area in a pre-defined template in order to 
determine the chosen option. 

15 

The method and the user interface thus provide a comfortable way for a user to control 
any device by simply aiming a compact hand-held pointing device to make an option 
selection without having to look away from the consumer electronics device while 
making a selection, and without first having to become acquainted with a multitude of 
20 buttons. 

The dependent claims and the subsequent description disclose particularly 
advantageous embodiments and features of the invention. 

25 User options for a consumer electronics device can be presented to the user in a number 
of ways, both static and dynamic. The simplest visual presentation of the user options 
for a device in static form is the front of the device itself, where various options are 
available in the form of buttons or knobs, for example the stop, fast forward, record and 
play buttons on a VCR. Another example of a static visual presentation might be to 

30 show the user options in printed form, for example as a computer printout, or a program 
guide in a TV magazine. Especially for a device such as a TV, or DVD player which 
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can be connected to a television, the options may be available to the user in static form 
as buttons on the front of the device, and can also easily be dynamically displayed on 
the television screen. Here, the options might be shown in the form of menu items or as 
icons. In a particularly preferred embodiment of the invention, user options for more 
5 than one device can be shown simultaneously in one visual presentation. For example, 
tuner options and DVD options might be displayed together, particularly options that 
are relevant to both devices. One example of such a combination of options might be to 
display a set of tuner audio options such as surround sound, Dolby etc, along with DVD 
options such as wide screen, sub-titles etc. The user can thus easily and quickly 
1 0 customise the options for both devices before commencing viewing. 

Another way of visually presenting the user options for a device in dynamic form might 
be to project them as an image backdrop onto a surface, for example a wall or a screen. 
Additionally, the invention might easily provide the user with a means of personalising 

15 the options for the device, for example by only displaying a small number of options on 
the screen at one time if the user has poor vision. Further, the user might specifically 
choose to omit functions that he is unlikely ever to require, for example, for his DVD 
player, he might never wish to view a film accompanied by foreign-language subtitles. 
In this case, he can personalise his user interface to omit these options from the visual 

20 presentation. A device such as a television can be configured so that for some users, 
only a subset of the available options is accessible. In this way, certain channels can be 
made accessible only by authorised users, for example to protect children from 
watching programs unsuitable to their age group. 

25 It is a central point of the invention that the user can choose among the options 

available by aiming a pointing device containing a camera at the visual presentation of 
the user options. The camera is preferably incorporated in the pointing device but might 
equally be mounted on the pointing device, and is preferably oriented in such a way that 
it generates images of the area in front of the pointing device targeted by the user. The 

30 image of the target area might be only a small subset of the entire visual presentation, it 
might cover the visual presentation in its entirety, or it might also include an area 
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surrounding the visual presentation. The size of the target area image in relation to the 
entire visual presentation might depend on the size of the visual presentation, the 
distance between me pointing device and the presentation, and on the capabilities of the 
camera itself The user might be positioned so that the pointing device is at some 
distance from the visual presentation, for example when the user is seated whilst 
watching television. Equally, the user might hold the pointing device quite close to the 
visual presentation, as might arise when the user is aiming the pointing device at a TV 
program guide in magazine form 



10 



In a preferred embodiment of the invention, a tight source might be mounted in or on 
the pointing device. The light source might serve to muminate the area at which the 
pointing device is aimed, so that the user can easily peruse the visual presentation even 
if the surroundings are dark. Equally, the light source might be a source of laser light 
15 source of a concenlrated beam of light emitted in the direction of pointing, so that a 
point of light appears at or near the target point on the visual presentation at which the 
user is aiming, providing visual positional feedback to help the user aim at the desired 
option. A simple realisation might be a laser light source incorporated in or mounted on 
the pointing device in an appropriate manner. In Hie following therefore, it is assumed - 
without limiting the invention in any way - that the source of concentrated tight is a 
laser beam. 



20 



For ease of use, me pointing device can be in the shape of a wand or pen in an 
elongated form that can be grasped comfortably by the user. The user can thus direct 
25 the pointing device at a target point in the visual presentation while positioned at a 
comfortable viewing distance from it Equally, tiie pointing device might be shaped in 
the form of a pistol. 

The pointing device might be aimed by the user at a particular option in a visual 
30 presentation, for example at the play button on the front of a VCR device, at a DVD 
option displayed on a TV screen, or at a selected program in a TV magazine. To 
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indicate that a particular selection is being made, the user might move the pointing 
device in a pre-defined manner over the visual presentation, for example by describing 
a loop or circular shape around the desired option. The user might move the pointing 
device through the air at a distance removed from visual presentation, or might move 
5 the pointing device directly over or very close to the visual presentation. Another way 
of indicating a particular option selection might be to aim the pointing device steadily at 
the option for a pre-defined length of time. Equally, the user might flick the pointing 
device across the visual presentation to indicate, for example, a return to normal 
program viewing after removing the visual presentation from the screen, or to a 

1 0 previous menu level. The movement of the pointing device relative to the visual 

presentation might preferably be detected by image processing software in the control 
unit, or might be detected by a motion sensor in the pointing device. A further 
possibility might be to press a button on the pointing device to indicate selection of the 
option at which the pointing device is aimed. In a preferred embodiment, the control 

1 5 unit can initiate a confirmation dialog in order to ascertain that it has correctly 

interpreted the user's actions, for example if the user has aimed at a point considerably 
removed from the optical centre of an option while pressing the button or moving the 
pointing device in a pre-defined manner. In this case the control unit might request 
confirmation before proceeding to initiate the selected option or function. 

20 

The control unit can preferably alter the visual presentation to highlight the selected 
option in some way, for example by making the option appear to flash or by 
highlighting the region in the visual presentation aimed at by the user perhaps 
accompanying this by an audible "click" sound. The pointing device might also select a 

25 function in the visual presentation using a "drag and drop" technique, particularly when 
the user must navigate through larger content spaces for example by dragging an icon 
representing buffered DVD movie data to another icon representing a trash can, thus 
indicating that the buffered data be deleted from memory. Various functions might be 
initiated by the user, whereby the user selects the option in a manner similar to a 

30 "double-click", for example, by repeating the motion of the pointing device in the pre- 
defined manner, or twice pressing a button on the pointing device. 
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In aparticularly advantageous embodiment of the invention, the image of the target 
area is transmitted from the pointing device in a wireless manner, for example using 
Bluetooth or 802.1 lb standards, to a control unit assigned to the device to be controlled. 
The control unit preferably comprises a receiver for receiving the target area images 
5 and an image interpreter, which might be in the form of a processor or controller. The 
target area images might equally be transmitted to the control unit by means of a cable 
attaching the pointing device to the control unit 

To determine which option has been selected by the user, the image interpreter 
10 compares the received target area images to a number of pre-defined templates of the 
visual presentation. A single pre-defined template might suffice for the comparison, or 
it may be necessary to apply more than one template in order to make a successful 
comparison. 

1 5 Pre-defined templates can be stored in an internal memory, or might equally be 

accessed from an external source. Preferably, the control unit comprises an accessing 
unit with an appropriate interface for obtaining pre-defined templates for the visual 
presentation of the device to be controlled from, for example, an internal or external 
memory, a memory stick, an intranet or the internet A template can b e a graphic 
20 representation of the front of the device to be controlled, for example a simplified 
representation of the front of a VCR device feataring the user options available, for 
example the buttons representing the play, fast-forward, rewind, stop and record 
functions. A template can also be a graphic representation of an options menu as 
displayed on a TV screen and might indicate the locations of the available device 
25 options associated with particular areas of the visual presentation For example, the user 
options for a DVD player such as play, fast-forward, sub-titles, language etc., can also 
be visually presented on the TV screen. The template can also depict the area around 
the visual presentation, for example it may include the housing of the device, and may 
even include some of the immediate surroundings of the device. 



30 



User options for a device which can display these on a screen can often be presented in 
the form of menus, where the user can traverse the menus to arrive at the desired option 
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or function. In a preferred embodiment of the invention, a template exists for each 
possible menu level for the device to be controlled, so that the user can aim the pointing 
device at any one of the available options at any level of control of the device. Another 
type of template might have the appearance of a TV program guide in a magazine. 
5 Here, templates for the layout of the pages in the TV guide might be obtained and/or 
updated by the accessing unit, for example on a daily or weekly basis. Preferably, the 
image interpretation software is compatible with the format of the TV guide pages. The 
templates preferably feature the positions on the pages of the various program options 
available to the user. The user might aim the pointing device over the visual 
1 0 presentation in the form of a page in an actual TV program guide to select a particular 
option, or the guide might be visually presented on the TV screen at which the user can 
aim the pointing device to choose between the options available. 

For processing the target area image in order to determine the chosen option, it is 
1 5 expedient to apply computer vision techniques to find the point in the visual 
presentation at which the user has aimed, i.e. the target point 

In one embodiment of the invention, a fixed point in the target area image, preferably 
the centre of the target area image, obtained by extending an imaginary line in the 
20 direction of the longitudinal axis of the pointing device to the visual presentation, might 
be used as the target point 

A method of processing the target area images of the visual presentation using 
computer vision algorithms might comprise detecting distinctive points in the target 

25 image and determining corresponding points in the template of the visual presentation, 
and developing a transformation for mapping the points in the target image to the 
corresponding points in the template. The distinctive points of the target area image 
might be points of the visual presentation, or might equally be points in the area 
surrounding the visual presentation, for example the comers of a television screen, or 

30 points belonging to an object in the vicinity of the device to be controlled and which are 
also recorded in the pre-defined templates. This transformation can then be used to 
determine the position and aspect of the pointing device relative to the visual 
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presentation so that the intersection point of an axis of the pointing device with the 
visual presentation can be located in the template. The position of this intersection in 
the template corresponds to the target point on the visual presentation, and can be used 
to easily determine which of the options has been targeted by the user. The position of 
the target point in the pre-defined template indicates the option selected by the user. In 
this way, comparing the target area image with the pre-defined template is restricted to 
identifying and comparing only salient points such as distinctive corner points. The 
term "comparing" as applicable in this invention is to be understood in a broad sense, 
i.e. by only comparing sufficient features in order to quickly identify the point at which 
the user is aiming. 

Another possible way of deterrnining the option selected by the user is to directly 
compare the received target area image, centred around the target point, with a pre- 
defined template to locate the point targeted in the visual presentation using methods 
such as pattern-matching. Another way of comparing the target area image with the pre- 
defined template restrict itself to identifying and comparing only salient points such as 
distinctive corner points. 

In a further embodiment of the invention, the location of the laser point, transmitted to 
the receiver in the control unit as part of the target area image, might be used as the 
target point to locate the option selected by the user. The laser point may be 
superimposed on the centre of the target area image, but might equally well be offset 
from the centre of the target area image. 

The control unit can be used to control a single device, but might equally be used to 
control a plurality of such devices, for example one control unit might be assigned to a 
TV, a DVD player, a tuner and a VCR. In this way, the pointing device can be used as a 
remote control for one device or for a plurality of devices. The pointing device can also 
be used to control a number of consumer electronics devices, each incorporating their 
own dedicated control units. Furthermore, a number of pointing devices can be assigned 
to a control unit, so that, for example, each member of a household has his own 
pointing 
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device. On the other hand, one pointing device might be assigned to a number of 
control units in different environments, for example so that a user might use his 
pointing device for controlling devices at home as well as in the office. 

In a preferred embodiment of the invention, the control unit might be connected to a 
projector which can project visual presentations of user options for a number of devices 
in the form of an image backdrop onto a suitable surface, for example a wall. The 
control unit might also avail of a separate screen, or might use a screen of one of the 
devices to be controlled. In this way, user options can be presented in a comfortable 
manner for a device which does not otherwise feature a display, for example a video 
recorder. The options represented by the buttons on the compact front of the device can, 
for example, be presented as menu options on the larger image backdrop for ease of 
selection. In a further preferred embodiment of the invention, the control unit can 
produce a hard-copy of a visual presentation, for example it can print out a program 
selection. 

Other objects and features of the present invention will become apparent from the 
following detailed descriptions considered in conjunction with the accompanying 
drawing. It is to be understood, however, that the drawings are designed solely for the 
purposes of illustration and not as a definition of the limits of the invention. 

Fig. 1 is a schematic diagram showing a user with a pointing device and a device to be 
controlled; 

Fig. 2 is a schematic diagram of a pointing device in accordance with an embodiment of 
the present invention; 

Fig. 3 is a schematic diagram of a visual presentation of a device to be controlled and 
an associated target area image; 

Fig. 4 is a schematic diagram of a device to be controlled comprising a control unit and 

a pointing device in accordance with an embodiment of the present invention; 

Fig. 5 is a schematic diagram of a control unit and a plurality of devices to be controlled 
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in accordance with an embodiment of the present invention; Fig. 6 is a schematic 
diagram showing a visual presentation and a corresponding target area image in 
accordance with an embodiment of the present invention. 

Fig. 1 shows a user 8 aiming a pointing device 2 at a device l a to be controlled, in this 
case a television l a . In this embodiment, the visual presentation 4 a of the user options is 
dynamically displayed on the television screen. The pointing device 2 generates images 
of areas targeted on the visual presentation 4. The television 1 contains a control unit 
which receives the images transmitted from the pointing device 2. 

Fig. 2 shows a pointing device 2 containing a camera 3, which generates images of the 
area in front of the pointing device 2 in the direction of pointing D. The pointing device 
has 2 features an elongated form, so that the direction of pointing D lies along the 
longitudinal axis of the pointing device 2. The camera 3 is positioned towards the front 
of the pointing device 2 so that images are generated of the area in front of the pointing 
device 2 at which the user 8 is aiming. The images are transmitted by means of a 
transmitter 14 enclosed in the housing of the pointing device 2 and are transmitted in a 
wireless manner, e.g. Bluetooth, or 802.11b standards A laser light source 7, mounted 
on the pointing device 2, emits a beam of laser light L parallel to the longiradinal axis 
of the pointing device 2 in the direction of pointing D. In this embodiment, the pointing 
device 2 features a button 1 1 . The button 1 1 can be pressed by the user, for example to 
confirm that he has made a selection and to transmit the image of the target area. 
Alternatively, such a button 1 1 might additionally be used to activate or deactivate 
displaying of the visual presentation, so that the user interface options are only 
displayed when actually required by the user. Alternatively, the function of the button 
1 1 might be to activate or deactivate the light source 7 mounted on the pointing device, 
or to activate or deactivate the pointing device 2 itself. Equally, the pointing device 
might be activated by means of a motion sensor incorporated in the pointing device 2, 
so that the visual presentation is automatically displayed when the user takes hold of the 
pointing device 2. The pointing device 2 draws its power from one or more batteries, 
not shown in the figure. Depending on the consumption of the pointing device, it may 
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be necessary to provide a cradle into which the pointing device 2 can be placed when 
not in use, to recharge the batteries. 

Fig. 3 shows a visual presentation 4 a - as displayed on a television screen or projected 
onto an appropriate backdrop - of a number of user options for a device to be 
controlled. A pointing device, not shown in the figure, is being aimed at a target area 6 
of the visual presentation 4 a . With the aid of a laser point P L generated by a laser light 
source on the pointing device, the user can select one of the menu options (Mi, M 2 , M 3 , 
M4) shown in the visual presentation 4 a . A camera in the pointing device generates an 
image 5 of the target area 6 centred around an image centre point P T . The laser point P L 
also appears in the target area image 5. The laser point P L may be a distance removed 
from the image centre point P T , or might coincide with the image centre point P T . 

Fig. 4 shows a consumer electronics device 1 incorporating a display, for example a 
television screen, and a control unit 16. The control unit 16 comprises a receiver 17 
which is capable of receiving target area images 5 of a visual presentation 4 a , shown in 
tins case on a screen of the device 1, transmitted from a pointing device 2 aimed at the 
visual presentation 4 a . The target area image data 18 is passed to an image interpreter 
13. An accessing unit 12 can access predefined templates, for example from an internal 
memory 19, an external memory 20, or the internet 21. Ideally the accessing unit 12 has 
a number of interfaces allowing access to external data 20, for example the user might 
provide pre-defined templates stored on a memory medium 20 such as floppy disk, CD 
or DVD. The templates may also be configured by the user, for example in a training 
session for the control unit 16, in which the user specifies the correlation between 
specific areas on the template with particular functions. 

The image interpreter 13 makes use of computer vision algorithms to compare the 
target area images 18 to the predefined templates in order to determine the option 
selected by the user. Information about the chosen option is forwarded to a dialog 
controller 10, which acts to instruct the device 1 to carry out the function associated 
with the option, e.g. to switch to the selected channel, and also instructs a visual 
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presentation controller 

9 which, for example, updates the visual presentation 4 a to show a different menu level 
or deactivates the visual presentation 4 a . 

5 

Fig. 5 shows a number of consumer electronics devices 1* H, U connected to a control 
unit 16. The devices l a , l b , lc feature different means of visually presenting to a user 
the options they proffer. Device l a , for example a television, displays its user options on 
its own screen 4 a . A user can aim a pointing device at the options displayed on the 
10 television screen 4* for example to switch channels, change loudspeaker volume, view 
videotext etc. The fronts of the devices l b and l c , for example a DVD player or VCR, 
display the user options for these devices and are used as visual presentations 4b and 4 C . 

Here, the user can aim the pointing device at one of l b or l c to, for example, fast- 
1 5 forward the DVD or video cassette contained in the device. 



Another visual presentation 4 e is shown in the form of a TV program guide l e . The user 
can aim a pointing device over the pages of the program guide 4 e to select a desired 
program. The control unit 1 6 then acts accordingly, for example by issuing commands 

20 to the television l a to switch to the appropriate channel if the program is already 

running. If the program commences at a later point in time, me control unit 16 might 
display appropriate prompts on the television screen 4 a , for example, whether the 
program is to be recorded later by the VCR l c . To ensure compatibility between image 
interpretation software and TV guide format, me software can be updated on a regular 

25 basis or as required by downloading up-to-date versions from the internet 21 . 

A beamer or projector 15 displays a further visual presentation 4 d in the form of an 
image backdrop, which might be associated with one or more of 1* lb, lc wim l e , or 
with another device not shown in the figure, onto a wall or other suitable surface. 
30 Images of target areas of any of the visual presentations 4 e transmitted from a pointing 
device (not shown in the figure) are received by the control unit 16 and compared to 
pre-defined templates of the visual presentations 4a, 4b, 4 W 4 d , 4 e . The predefined 
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ten5>lates can be stored locally in the control unit For example, predefined templates 

for devices l b and l c can be stored once in the control unit, since the front of such a 
device remains static. Templates for the visual presentations for a device such as a 
5 television l a might change intermittently, for example it might be possible for a user to 
program the device to customise the menu-driven operation, so that the menu levels and 
their corresponding templates might take on a different appearance. The templates for a 
TV program guide are updated by the control unit with every issue of the program 
guide. These templates are downloaded from the internet 21 by means of a suitable 
1 0 interface. The control unit 1 6 determines which of the visual presentations 4 a , 4b, 4^ 4 d , 
4 e has been targeted, and which option has been selected The control unit then issues 
appropriate commands to the devices in order to carry out the requested function. 

Figure 6 shows a schematic representation of a target area image 5 generated by a 
1 5 pointing device 2 and a template for the visual presentation 4 a . The pointing device is 
aimed at the visual presentation from a distance and at an oblique angle, so that the 
scale and perspective of the elements (M i, M 2 , M 3 ) in the visual presentation 4 a appear 
distorted in the target area image 5. To determine the option selected by the user, in this 
embodiment the point of intersection P T of the longitudinal axis of the pointing device 2 
20 with the visual presentation 4 a is located. The point in the template corresponding to 
the point of intersection P T can then be located to determine the chosen option. To this 
end, computer vision algorithms using edge- and comer detection methods are applied 
to locate points in the target area image [(xa, y a ), (x b , y b ), (xc, y c )] which correspond to 
points in the template [(x a f , y a '), (x b ', y h % (x^, y c ')] of the visual presentation 4 a . 

25 

Each point can expressed as a vector e.g. the point (Xa, y a ) can be expressed as v a . As a 

next step, a transformation function T\ is developed to map the target area image to the 
template: 

fW=Z|TxC? i )-v;| 2 

i 

30 where the vector v { represents the coordinate pair (xj, yj) in the target area image, and 
the vector v[ represents the corresponding coordinate pair (x' is y^) in the template. The 
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parameter set X, comprising parameters for rotation and translation of the image 

yielding the most cost-effective solution to the function, can be applied to determine the 
position and orientation of the pointing device 2 with respect to the visual presentation 
5 4*. The computer vision algorithms make use of the fact mat Ihe camera 3 within the 
pointing device 2 is fixed and "looking" in the direction of the pointing gesture. The 
next step is to calculate the point of intersection of the longitudinal axis of the pointing 
device 2 in the direction of pointing D with Ihe plane of the visual presentation 4 a . This 
point may be taken to be the centre of the target area image P T , or, if the device has a 
10 laser pointer, the laser point P L can be used instead. Once the coordinates of the point of 
intersection have been calculated, it is a simple matter to locate this point in the 
template of the visual presentation 4 a , thus determining the option which the user has 
selected. 

15 Although the present invention has been disclosed in the form of preferred 

embodiments and variations thereon, it will be understood that numerous additional 
modifications and variations could be made thereto without departing from the scope of 
me invention. The remote control pointing device can serve as the universal user 
interface device in the home or for navigation through business presentations. In short, 

20 it can be beneficial whenever an intention of a user can be expressed by pointing, which 
means that it can be used for essentially any kind of user interface. Its small form factor 
and its convenient and intuitive usage can elevate such a simple pointing device to a 
powerful universal remote control. Its ability to be used to control a multitude of 
devices, providing access to content items of the devices, as well as allowing for 

25 personalisation of the device's user interface options, make this a powerful tool. As an 
alternative to the pen shape, the pointing device could for example also be a personal 
digital assistant (PDA) with a built-in camera, or a mobile phone with a built-in camera. 
The pointing device might be combined with other traditional remote control features, 
e.g. with additional buttons for performing dedicated functions such as fast scrolling 
30 through a long fist of items, or with other input modalities such as voice control for 
direct access to content items of the device to be controlled. 
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The usefulness of the pointing device need not be restricted to the control of consumer 
electronics devices, for example it may equally find application within a medical 
environment, in a home or office environment, or in industry. For example in a home 
environment, such a pointing device could be used to control almost any electrical 
5 appliance which can be controlled by a control unit, for example washing machine, 
cooker, heating, light fittings. Such a pointing device could make life considerably 
easier for users who are handicapped or so restricted in their mobility that they are 
unable to reach the appliances or to operate them in the usual manner. 

1 0 For the sake of clarity, it is also to be understood that the use of "a" or "an" throughout 
this application does not exclude a plurality, and "comprising" does not exclude other 
steps or elements. A "unit" may comprise a number of blocks or devices, unless 
explicitly described as a single entity. 
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CLAIMS 



LA method for control of a device (l a , H, U), which method comprises: 

- visually presenting a number of user options for the device to be controlled 

(la, lb, lc); 

- aimin g a pointing device (2) comprising a camera (3) at the visual 
presentation (4 a , 4b, 4 C , 4 d , 4 e ) of the user options to choose a desired option; 

- generating an image (5) of a target area (6) aimed at by the pointing device 
(2); 

- comparing the target area image (5) with a pre-defined template of the visual 
presentation (4 a , 4b, 4 C , 4 d , 4 e ) to determine the chosen option. 

2. A method according to claim 1, where a source of a concentrated beam of light (7) 
attached to the pointing device (2) shows the user (8) a light point (P L ) in the visual 
presentation (4 a , 4 b , 4 C , 4 d , 4 e ) at which the pointing device (2) is aimed. 

3. A method according to claim 1 or claim 2, where the chosen option is determined by 
locating a point in the template corresponding to a target point in the visual presentation 
(4 a , 4b, 4 C , 4 d , 4 e ) at which the user (8) has aimed the pointing device (2). 

4. A method according to claim 3, where the light point <P L ) is located in the target area 
image (5) and is taken to be the target point. 

5. A method according to claim 3, where a fixed point (P T ) in the target area image (5) 
is taken to be the target point 
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6. A method according to any of claims 1 to 5, where a desired option is selected by the 
user (8) by aiming the pointing device (2) at the desired option in the visual 
presentation (4 a , 4b, 4 C , 4 d , 4 e ) and pressing a button (1 1) on the pointing device (2). 

5 7. A method according to any of claims 1 to 6, where the desired option is selected by 
the user (8) by moving the pointing device (2) over the visual presentation (4 a , 4b, 4 C , 
4 d , 4 e ) in a pre-defined pattern. 

8. A method according to any of claims 1 to 7, where the target point is determined 
1 0 using computer vision algorithms. 

9. A method of any of claims 1 to 8, where the target point is determined by a method 
comprising the following steps: 

- detecting distinctive points in the target image (5) of the visual presentation 
15 (4 a ,4 b3 4 c ,4 d5 4 e ); 

- determining corresponding points in the template of the visual presentation 
(4 a , 4b, 4 C , 4 d , 4 e ); 

- developing a transformation for mapping the points in the target image (5) to 
the corresponding points in the template; 

20 - using the transformation to determine the position and aspect of the pointing 

device (2) relative to the visual presentation (4 a , 4b, 4 C , 4* 4 e ); 

- locating the intersection point of a certain axis of the pointing device (2) 
with the visual presentation (4a, 4b, 4 C , 4 d , 4 C ). 

25 1 0. A method according to any of claims 1 to 9, where the visual presentation of the 
device options (4b, 4 C , 4 e ) is presented in static form. 



1 1 . A method according to any of claims 1 to 10, where the visual presentation of the 
device options (4 a , 4 d ) is presented dynamically. 
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12. A method according to any one of claims 1 to 11, where one or more target area 
images (5) of user options for a plurality of devices to be controlled (la, lb, lc) are 
generated and compared to pre-defined templates and, depending on the option chosen, 

5 one or more of the plurality of devices (la, lt>, lc) are controlled accordingly. 

13. A user interface for control of a device (l a , lb, lc), said user interfece comprising: 

- an accessing unit (12) for accessing pre-defined templates associated with 
visual presentations of user options for the device to be controlled (la, lb, 

10 lc); 

- a pointing device (2) for aiming at a desired option in a visual presentation 
(4 a , 4 b , 4 C , 4 d , 4e) of the user options, comprising a camera (3) for generating 
an image (5) of a target area (6) of at least part of the visual presentation (4 a , 
4b, 4c, 4 d , 4 e ); 

15 - an image interpreter (1 3) for locating the target area (6) or a point of the 

target area (6) in a pre-defined template in order to determine the chosen 
option. 

14. A user interface according to claim 13, comprising a transmission interfece (14) for 
20 transmitting the images (5) to a control unit (16) assigned to a device (l a , lb, lc); 

15. A user interface according to claim 13 or 14, comprising a display unit (15) for 
dynamically displaying a visual presentation (4 d ) of the user options for the device to be 
controlled (l a , lb, lc)- 

25 

1 6. A user interface according to any of claims 1 3 to 1 5, comprising a hardcopy output 
unit/module for generating a static visual presentation of the user options for the device 
to be controlled (l a , lb, lc)- 
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17. A pointing device (2) for a user interface according to one of claims 13 to 1 6, 
containing a camera (3) for generating an image (5) of a target area (6) in the direction 
(D) in which the pointing device (2) is aimed. 

5 1 8. A pointing device (2) according to claim 17, comprising a light source (7) for 
illuminating the target area (6) at which the pointing device (2) is aimed. 

19. A pointing device (2), extending along a longitudinal axis, containing a camera (3) 
positioned in the pointing device (2) such that the camera (3) generates an image (5) of 

10 a target area (6) in front of the pointing device (2) in the direction (D), along the 

longitudinal axis of the pointing device (2), in which the pointing device (2) is aimed. 

20. A control unit (16) comprising a receiver (17) for receiving target area images (5) 
from a pointing device (2), an accessing unit (12) for accessing predefined templates 

1 5 associated with visual presentations (4* 4^ 4c 4a, 4 e ) of user options for a device to be 
controlled (l a , lb, l c ), and an image interpreter (13) for locating the target area (5) or a 
point of the target area (5) in a pre-defined template in order to determine the chosen 
option. 

20 21. A device (l a ) comprising a control unit (16) according to claim 20. 
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ABSTRACT 

Method and system for control of a device. 

The invention describes a method for control of a device (l a , lb, l c ). The method 
comprises visually presenting a number of user options for the device to be controlled 
(la, lb, lc), aiming a pointing device (2) comprising a camera (3) at the visual 
presentation (4 a , 4b, 4 C , 4 d , 4 e ) of the user options to choose a desired option, generating 
an image (5) of a target area (6) aimed at by the pointing device (2) and comparing the 
target area image (5) with a pre-defined template of the visual presentation (4 a , 4b, 4 C , 
4«j, 4 e ) to determine the chosen option. 
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